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A' Pr*j|U'ain Toi" Kstlina Mn;^, the Kelath r'fficlency of T^-sts at 
V'li'l'jus Ability ].f'V^.-.l.L*,^ for Rqiiating Tfuc- Seofes, and for 
Predicting BivariaLe Distributions of Observed Scores'^ 

For two tests measuring the same trait^ the ])rogram^ fiTVr.^O^ equates 
th( ^;r"ie scores using the two true-score distributions estimated by the 
un iv-j I'late method. JO program (see Wingersky, Lees, Lennon, &: Lord^ 'iy69) 
andV^ 'with these, equated true scores and their distributions, estimates 
u\" b'varlrite distribution of observed scores and the relative efficiency 
of til' two tests at various ability levels. Tlie" method ,is described and 
form\las ;';iven in Appendix A. If desired, this estimated distribution is 
conxj -xred to an. actual bivariate observed-score distribution provided by the 
user, 'ind a chi-square between the estimated and actual distributions is 
-Gom]'-iL^-d. 

rhin j-ro^'^ram is wi-LLten in Fortran IV for the IBM 360/6>. Its only 
res'.i-ict ion 13- that the maximum number of Items for each test is 

' 'Sec , " - 

Gom- r the uses of the ]To^;;ram arc- tho foIlr:)\ring (s-ee Lord, 1S^5^ for 
details): . 

1. T'j chrek on the mthematical modrJ used to estimate the true- 
score distributions ("Locd, ^ ■ . ■- 

^'Thi.s updates the search [■*• morandum (vw-,;"^ titled "A program i?or 
estimating a bivariate- distribwtlon of test . scores from the marginals" by 
"Martha Stocking, Diana Lees, ' VJ rgi nla Lennon and Frederic M. Lord 
suj^j>ort(/d in-part by conti'act ilonr-?J^2(00) between the Office of Naval 
i-;esearch and Educational T(-::,ting Service. A.dditions and revisions to the' 
3)rogram were supported b;; grant <"iB-:>278L>' ■ from National Science'^ foundation. 



1^ To estimate how a group of examinees that has taken only a short 

form of a test vouid have performed on a long form. (This, is 

useful in setting up test norms.) 
'3- To estimate hovr a group of examinees that has been selected on 

a certain test v/puld perfortQ on a parallel .test. 
^. To try to equate two rather different tests Treasuring the same 
/ trait wheo both tests cannot be given to the same group of 

examijnees. , 

5- To investigate whether or not two different tests measure the 
same psychological trait. (if tne program does not provide a 
" good estimate o.f the actual biyariate- scatterplot, it may be 
possible to conclude that the assumption—that the tv;o tests 
measure the samt /'Syohological trait has been violated.) 
6. To compute "the relative efficiency of the.tvro te-si^ at various 

ability levels. • ■ 
An illustration equating true, scores and estimating relative 
efficiencies for several pairs of tests is giv^n in L )rd^ 1973b* . 

A.S sumptions I . t • 

The results are based on the following 'Assumptions (see Lord^ 196?^ fo^ 
a' detailed and rigorous statem<-nt): 

i. The conditional distribution of observed scores for fixed true 
score is a (certain approximation to a) ■ compound binomial 
dlsti'ibution. 
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> 

The true-score distribution in the group tested is ^smooth." 
'j. The true, scores on the two tests studied are perfectly/' 
•although possibly curvilinearly, related. 

nput . . . 

. The output from a univariate method-20 program is required for both 
(.jLo bein;7 investigated. For detailed description of the input fortnats^ 

reader is referred to Appendix 'c. 

. t» . ^ 

.It ■ • • ■ ■ 

The user receives as printed output the following: 

1. The estimated bivariate observed-score distribution and the 

regression of each 'eBt lmated test on. the other (row and column 

means of the estimated bivariate observed-score distribution). 

ri.. i^he estimated, marginal . observi?d-SGore distributions; the mean, 

varian'cii, and K.wder-;;iohardson iv,-^ for each of these distiri^u- 

' ti'Mis; also for' the- estimated bivariate distribution, the 

. estima.ted corrt'lation betv/een observed scores and the cor- 

rr-Lation ratios fo:' Test X c^-ven Test Y and Test Y given 

Tect 7. ^ ■ . . ■ . • ■ 

'J*' The/estimated probability distribution of true scores for each 

e o t . ~ "' . 

An equatin-^ of. the true scores of botlr tests by the equiperc entile 

method- • \ 

>. The fT-latlve' erfi\::ier.icles 'of the two tests at various ability 

^ 'levels. ' . . ■ . * 



Ii* the actual observed-score bivariate distribution is provided, the 
user receives the additional output: . ' 

6. The chi-scjuare between the estimated and actual bivariate 
ob^erVed- score distributions, and its probability level; the 

. grouping of the two bivariate disisributions for the chi-square, 
the estimated and actdal group frequencies,- and the group con- ^ 
tributions to chi -square, e " 

7. A graph; showing for each cell in the bivariate distribution 

a if the estitnated frequency for the cell was greater than 

2 

1, also the si^.n of the contribution to the X of the group 
containing the cell if the. contribution v/ns ^^rua;■el• in absolute 
value than 2. . 

cj. The regression of each actual test on the other (row and column -^^ 
means cf actual bivariate observed-score distribution)^ These 
can >be compared with item 1 above. 
9- tTie actual marginal observed-score distributions; the mean-^- - 
variance, and Kuder-iuchardson li for each of these distri- 
butions, the actual coi^relation between the observed scores, *^ 
and the -correlation ratio for Test X given Test Y and for 
Test Y given Test X for the observed bivariate distribution. ^ 
These may be compared^ v;ith ' item- 2 above. 
The program planches the logarithms of the estimated relative 
efficiencies, the .-crue-score percentij.es at which they are computed and 
the estimated and actual marginal distributions for both tests along 
v/ith identifying info i:mat ion. ^Fhe equated true scores may also be punched. 



A va liability ^ - ' V 

A cop:;. of tlu^- prosram may be obtained upon VrrittGn roqucpt frotn --fehe 
aiithui's* llducational Testing Service, Princeton, New Jersey, 08s^0« The 
user must I'rovJde a tape on -wiiich the progratn will be loaded in 8Q 
chara.!tfi^ card iniages and must specify whether the tape should be blocked, 
in"*]'BCDrC. or BCD;, .J-track or 9-track, and the, tape density and parity. 

The. tape vrill be unlabeled. ' . • 

^ .. • - ■ , ■ 

Disclaimer ' • \ ■ " 

Althoitgh the program has worked satisfactorily on-^he isjta we have 

« ■ . f 

tried, nc claim is made that the program is . free of error. 



Appendix A 

'( 

■ . ■• ■ \ ■ ' 

The program BIV20 computes the estimated bivariate distribution of ob- 

served scores between two tests; Test X and Test using the parameters.-9f — 
the corresponding univaric^te true-score distributions; If we let ^I^C^^y) 
be - the population bivariate distribution of observed scores for the two 
tests^ be, the true score on Test g(0 distribution of true 

.^:*ores Tor Test X, h(x,y|r) be the joint conditional distribution of ob- 
servcii scores x and y on Test: X and Test given- ^ , and a^ and 

be the lower and upper limits respectively, of the' true-score distribu- 
tion for Test X; then 

(1) '»(x,y) = I g(r,)h(x,yk) d? " ■ ■ ' 

a .. ■ ' ' 

If v/e let T, be a true score on Test Y and g'C^) t>e the distribution of ; 
true scores for Test Y, then under assumption 5, t] is a function of ^ . 

"When (i( ) and g'(r|)- ■a:^^gl^e^j~^i,hi'&~f^^^ — ^yy 

can be determined numerically^ for any fixed , from 

(2) f g(0 dr = I g'(,,) <i,, ■ ^ - ■• 

■ . ' . a . ■ a . - " . ' 

■ ^ :.l \ y ■ ■ , ■ 

v;here a., i.: Uie lower lirait of^the tt-ue-score ddstribution for Test by 
invers^' ihterpolation methd is. Under tlie assumption, that the errors of 

'-measurement -are in iepen-iently iistributed • v/hen r is fixed, equation ' (l) ' 

• tjecomes . . ' • 

b,-'^ • V ' _ ■ V. . ■.. -. ■ ■ ■'■ • 

(5) ; 4)(x,y) /" .g(r)h(x[Oh'.(yjTi) dr ' * , • " ^ ■' 



' >7here h(x|t,) is the conditional distribution of observed score x on 
Test X given a fixed tx'-ue score r< ^ and h*(y|i])' is the conditional 
distribution of observed score y on Tast Y for a fixed true score 'T| 
.where ij = \lr(r;) . If we let denote estimated quantities, then 
equation (2) becomes ■ < ' . * 

• and .equation (5) becomes " 




The estimated bivariate distribution of observed scores, $C>^^y) j is ob-' 
■tained By first determining frpm equation t-hen- performing the • 

integration indicated in equation ' (^ ) ' * by quadrature. ' ..' ^7 

■ The reader is referred, to equation L'i (Lord, 1967) for' the matheTnatical 
form of g(0 and g'C^i) , and to equation 7 (Lord Lees, I967) for the 
mathematical form^ of ■ h(x|c) and h*(y|ij)\ 

The estimated relative" efficiency of Test Y versus Test X is given by 
the formula. - 



where k and k are defined in' Lord, 1965/ eq. 9. n. is thfe nuffiber 
X y , X' 

of items on Test X and n is the n amber of items on Test Y.- For the 

_ ■ y 

'derivation of this formula see Lord (i975a). 'The relative efficiencies 

are computed for equally spaced T 's' between and . A^raph is 

produced of G(r) vs log,^ R.E. where 
^ . " ^10 yx 

for the equally spaced C 's- 



. ' ■ Appendix B ' ^ . . . 
Accuracy 'of Com putations' 

• — <j ^ 

■ As a general role the user obtains e-stimated marginal- observed-score 

distributions- which agree to^at least .three dedimal places with^-th-e— - — - — 
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estimated observed-scpre distributions obtained f rom a ^un-i:\rariate method-20 
pro^^rarn. There- are .two major soj^ces of numerical error. 

c . . ■ ■ > ■ 

1. The numerical solution of equation 2/ Appendixf^A. This solution 
is accomplished "by an iterative inverse interpolation procedure using 
Bessel's interpolation formula. (Scarborough/ "1955; P* ^7); ^ndy :as with 
anv' . interpolation process^, this generates computational errors." 

The ac'cura'cy attempted by this iteratdve procedure-is controlled ^ 
by an input option available to the user. - ■ • ' 

.2. Fiquation 3 of Appendiy A is inte.grated" with Simpson's rule 
whore the us'^^r specifies th'e number of intervals. Accuracy can be in- 
crf^ased by using more intervals.""' Hovrever^ from p^ast experience' approxi- 
mately 50 invervals Have been sufficient. " , • * 



For true -score distributions' where »gC^. = 3-nd/or g((^ = b^) 

are not close to 0^ the error in' integrating the corresponding tail .of 
/■•(x^y) can be quite large. There are two ways to possibly reduce " } 
this error. The user may specify that the Simpson's rule interval con- 
. tarn-_Lng a^ and the inter-v:al containing b^. be subdivided into, much 
.. smaller intervals for a more accurate, computation of the, area under' the 



ERIC 
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0 / 



■ % 
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curve in these intervals. Also it may be possible to increase thq 



accuracy-by reversing' the abscissa.and ordrna1:e , .put ting Test Y. "on 

.the abscissa and Test X on^ the ordinate / T This vill only help ift for 
Test Y', .g(r|- = a^)' and g(,n = • approximat-t^ly 0. ' » *^ 

The maximim number of intervals plus subintervals in the tails 



as 280. 



ERIC 



-11- 



Appendix C 

•input to Bivariate Program for Method-20 (BIV20) 

'In the following description of_ the input -required by BIV20^ the 

test on the abscissa of the bivT^riate': scatterplot is referred to as "Test"-- 

\ 

y." The test on the ordinate of tlae bivariate scatterplot is referred to 
as "Test Y." Zeta (^) denotes the'\^rue score for Test and eta (r\) 
denotes the true score for Test Y." All input information marked with a.n ^ . 
asterisk (*) coraes from the univs-riate method- T' xtput. All other- infor- 
mation must be supplied by the user. All iil,:-- v information followed .by a • 
(-t) can be continued on more than one card. For detailed descriptions 
of the mathematical functions for .the input .variables^ the user is 

■ referred to Lord and Lees (1967) and Lord (19$?)^ • 

Card Number Description 

■ • ^1. Title card for Test X : 

col. 1 - .60 title of tect| to be used as heading 
information on .output. ■■ 

col^63 - 65 ■* nutrfcer of items in test^ must be 
leas, than 'or equal to ^0*- The 
format is (I?). 

col. 66 ■- 72 ' number of .e:x:aminees feking test. 
The format is (FT-O). • 

.^2'. ■ Title card for Test Y vfith the' same information and 

. format as card number i; . , " 

*5. Parameter card for Test X. These parameters are tiecessary. 

to compute the estimated. true -score distribution fO,r 
Test X. The format, is (5E15-8).- ' \.. 

■ 'i : . . ■ ■ - ' • ■ . \ 



col. 16 -■ 50 
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Card Numbe r D escription 

"^•J- (contM) ' .col. 51-^5 ^ pa'>:-aineiers for the "smoothing 

function^^ 7(0 d.eboribe(i 
col. ^6 - Go in J.urd, I969. | 

I ■* 

col. 61 - 75 



X 



(if thd' simj>lified nEthod-20 program has been used, 
a - 0 , b = 1 d. -^A^ = 0 . ) 

''^h. Parameter card for Test Y with the same information 

and forirdt us card number 3- 

5- Options Card • . , • . 

col. 1 - 5 , punch the number of quadrature 

intervals toie used in computing 
<t>(x^y) . This number must be 
less ^than or equal to 280, and 
must be even. Generally this 
number ranges between 5O ^-nd 100. 

col. 6 - 10 punch 0 ' if the -^s and r] ' s 

are to be equated by the 
program. This is the 
usual, option. 

c 

1 

, 1 if equated' (; 's and t) 's 
are supplied by the user. 
The user must then pro- 
vide cards number 15 (b)+. 

-1 if the ^ /s and t) 's 

. . are assumed to- be identical. 

col. .11 - 15 punch 0 if the grouping for the » • - 

chi-square is to be 
^ ■ computed by the .program^ 

This is .the usual option. 

1 if the grouping for "^-he " 
chi'-square is tc be com- 
puted by the program, 
and vrritten on scratch . 
tape k. 



2 



if the grouping for the 
chi-square is. to be read 
from tape unit h. ^ 
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Card Number 



:')• (cpnt^d) 



Description 



col. 11 - 15 
(contM) 



if the grouping for the 
chi-square is to be 
read from cards. The 
user must provide car(?[s 
number 19^ nui.l3er 20+^ 
number 21+. 



col. 16 - 20 



*8. 



col. ::i - 25 



punch 0 if equated true scores 
are not to be punched 
" . on cards. Tliis is the 
usual option. 

1 if 'equated true ^scores 
are to be punched on 
catrds. The format will 
be (5E15-8)- 

punch the number of subintervals 
to-be used in integrating the 
. interval containing a 



interval" containing b 



and the 
Two 



times this number plus thie con- 
tents of cols. 1-5 must be less^ - 
than 280.' 

Parameters required for the conditional distributions 
of observed sco-res for a given true score for Test X. 
If the generaL univariate method--^20 -prog-ra-m -has-been - 
used^ the^e are part of its output-. If the sim[E)lified 
version has been used^ these values may be obtained 
from Lord and Lees (1967)^ rFigures 10 and 11. , The . 
format is (2E15..8). 

- col. 1-15 '-.i 

coi. 16 - 30 3- 

The same information and format as cafd number 6 for 
Test V. * ■ . 

The const'ants associated with the smoothing/ functions 
/(j:, ) and 7(rj) required to estimate the true-score 
distributions for Test X and Test Y. These constants 
are the reciprocals of ^ \ 



Card Number Description 
*8. (cont^d) 



. b 
f A d A 

/ r '(1 - ^) ^ dC for Test X, and ./ '»(r ^ ^ 



' dr, 



for Test Y. If the general'- univariate method-20 pro - 
■ gratn has been used^ these- are part of the o.u"Cput. T± 
the siirrplified version has been used^ both these con- 
stants are equal to 1. The format is (2E15-8). 

■ ■ . ^ . ' '■ . 

col. 1 - 15 7(0 constant. 

■ col. 16-50 vi^) constjant. 

-*9. ^ The coarse ^grouping of frequencies of observed scores 

used in estimating the true -score distribution of T 
' . Test X. • The format Is (k0l2).- . ■ . 

col. 1-2' the number -,of coarse groups. 

This number must be less than 
■ or equal to -26. 







col. -j 


. k \ 


1 for each group^ the number of 




r' 


col. [; 




frequencies in that group.. ■ The 
listing, of groups- starts wit);! 






■.col. 7 


-8 ^ ( 


the group containing the fre- - 










> quency for the .levies t observed 




4>- 


col. 


- lOc 


score^ and ends wi"Ch the group 
r containing the frequency of the 
1 highest observed score- 



*10. The same information and format as card number 9 fo^^.. 

Test Y. ■ . ' ' . 

•^11. The parameters - of the true-score distribution 

for Test X. The format is (5E15.8-% 

■^12'. The same information and format as card number 11. ' 

• for Test. Y. - ■ ' 
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Card Number Description 

15. (a) [f equated true scores are to be. conrouteJ by the pro- 

-gram (col. 10 of card number 5 equals O)^ the epsilon 
for the iterative inverse interpolation process is used 
, ' to equate true scores. If the results of two -sue-. 

■ ceS'Sive 'iterations differ by less than epsilon/ the 
- ■■ iterative procedure is halted. This, epsilon is . 

V usually .00001.. The format is (F15-8). 

(b)+ ■ , If equated true scores are to be supplied by the" user 
(eol. 10 of card number 5 equals l), .the true scores 
for Test X. followed (starting on anew card) by the 
true scores for Test Y.' The format mast be .(li^F5.5)- 
There must be the same number o'f 's and t) *s^ •„ 
and this number must be one greater tha-n the nuinber 
" ' ■ of quadi-ature intervals given in col.. 1-5 of ' card 
number 5* ' 

Note that 13 (a) and 15 (b )+ are 'mutually exclusive. 

Ik. Criteria for the grouping of the estims.ted and actual 

•\ bivariate observed^score distributions -for the ehi- 

square. (Note the information required in col. ^7' 
kQ before f)roceeding. } ■ The format is (2F&. 3^612). 

col.' l - 8 punch the minimum group size ^ 

required. Usually if the number 
■ of examinees is greater than 
3000 the minimum is 30; if it ■ 
■ is less than or equal to 3000^. 
the minimum is 20« ^ 

col. 9 - 16 punch the minimum cell frequency 
- for the -estimated bivariate 

observed-score distribution. All 
cells v/ith a frequency less than 
* • , or equal to this value will be 

set to zero before the grouping 
: " . ■ for chi -square is done. This 
' . . ' minimum is usually .005- 

col. 17 - hC . 'punchy in (1512) format^ the 

• following numbers: 00^01^02/05/ 

' ■ 08,09,10,13,16,17,18,20,25,285 

4 ' .32. These are - distance criTbei^ia 

. . ' </ ' used to determine whether a 
" ■ ■ ^prospective cell should be 

o " - , included in a group. 
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Card Number Descri]^tion 

l^f. (cont'd-) col. k'J' - kd punch 0 if the act^jal bivariate 

observed-score distri- 
bution will be read; 
and the chi -square 
*■ ' ' computed. The user 

must provide cards 
• . , number 15; number 16+^ 

\ n^jmber 17; nuinb.er l8+. 

!■ ' ' . \ 1 if the actual bivariate 

f ^ • observed-score distribu- 

tion will not .be read. ' 
Col. 1-46 may be left 
blank. The program will 
■ . halt after the computa- 

tion of the estimated 
bivariate observed-score 
distribution. 

■15. If the actual bivariate observed-score distribution 

■ is to be read by the prog:i;;am (col. ^7-^8 of card 
" number 1^+ is zero) punch -this card and cards number 
number 17> and,^number l8+, 

• * col. 1"- 30 the integer format of the actual 

. . bivariate observed-score distri- 

. ^ but ion which must be read in by 

' .rovsy with each row being' the 

■ ^ ' ""frequency "^distribution of ob- 

„ * served .scores on Test (in 

■ . ,;r ascending order by , observed 
- . f ' score) for a given obs'ferved 

' ■ ^ score on Test X. For example^ , 

(201^ y punched in these columns 
- would mean that the actual b j - 
variate scatterplot will be 
punched with- 20 four-'digit 
'- ^ intege-rs per .ncard. • , . . ^ 

- ^ col. 51 - 35 "the nuMDer of elements per . card. ' 

• . • ^ the forma Vis (I5)* 

w col. 56 - 'kO 'the observed score on Test' X .for 
' " ■ - ' ■ the first row to be ^read in 

■ ' ' . . ' (usually DJ. Rows must be read 

' ' • ' . in ascending order of '.observed - 

' ^ " .scores on. Test X. This means 

^ ' ■ ° . o that initial "^rows of/all '-z-eroes 

"need not .be punched^ ' but. will 'be 
■ ' ■ . ' assumed 'zero by the. program. The 

formt.'- is.'(l5).. ■ 



Descriptibri ' 

col. kX - 



Note: This option is used only 
... ..... ^- ■ ■wllen coi. 56-^0 is zero. The 

■ format is' (l-5X.- ' . ^, 

Actual bivariate scatterplot^ punched by. rows in the; 
forinat. given by card, number 15* There are .two ways : 
in which the s'catterplot "may be punchecij depending^ \ 
upon .col. t>6-hO of card number 15* • |* 

If col.' 36-40 cf card number 15 is not equal to zero^ 
consecutiv-a zero cells may be replaced by a' negative 
nuiT^er whose magnitude equals the number of zero 
cells to be omitted. Each neV row does not start 
'o*n a' new- card. ■ 

If col. 56 - 40 of card number • 15 equals, zero^ all. 
consecutive zero cells must be punched^ and each new 
row must start a . new -card-. . " ■ ^ 

Punch in ool. .1 7 50 the integer forrftat of the actual 
marginal frequency distributions which will be read. 

Actual marginal frequency distribution qf Test 
followed' (on a new cal^d) by that of Test Y^ both 
punched in- the format; given -by card number 17* Each . 
marginal 'frequency distribution is punched in 
ascending;' ord-fer of observed score. 

If the grouping for the chi-square is to be read 'from 
cards (col. 15 04 card. number -J is equal to number 3)j 
punch' the integer forrrtat * in which the grouping, will 
be read in col. ■.1-601 The i^ser must also- punch cards 
number 1:3+ . and 'nurnber 21+. ' (The information in col. 
1-46' of card number** l4 v/ill be ignored.) . — 

Matrix of group . numbers^ one for each cell in the bi-^ 
variate scatterplot^ punched by rows in the format 
given by card number 19, with each- new row- starting 
a nev/ card. The . ij -th element of this matrix is 
the number of the group -to v/hich the ij,- -th cell 
of the bivariate scatterplot belongs'. 



punch 0 if the Input matrix has 
/ not been transposed. 



1 if the input mat^rix has 
^ 'been transposed. . 
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Card Number 



Description 



Information required for the combining of undersized 
groups^ punched in the integer format given by card 
number 19. 



punch first the total number ^of -groups 



second the numiber of groups to be combined 

third ■ old group number^ 
new .group numiber 



old group number^ 
new group number *l 

old grpup number^ 
new group numiber 



for each group 
to'be movedjj 
starting with 
the^ lowe-st 
oTd^roup 
number 
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